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ON LIPSCHITZ CONTINUITY OF VALUE FUNCTIONS 
FOR INFINITE HORIZON PROBLEM 

DMITRY KHLOPIN 


Abstract. We investigate conditions of optimality for an infinite- 
horizon control problem and consider their correspondence with the 
value function. Assuming Lipschitz continuity of the value function, 
we prove that sensitivity relations plus the normal form version of the 
Pontryagin Maximum Principle is a necessary and sufficient condition 
for the optimality criteria that correspond to this value function. Dif¬ 
ferent criteria of optimality under different asymptotic constraints may 
be used, including almost strong and classical optimality proposed by 
D.Bogusz. Special attention is devoted to weakly agreeable criteria. We 
also obtain the conditions on control system (like controllability) that 
guarantee the Lipschitz continuity of the value function, without any 
other asymptotic conditions besides finiteness of the value function. 

Some examples are discussed. In particular, it was shown that the 
same control, regarded as agreeable optimal and overtaking optimal con¬ 
trol, can correspond to different (everywhere) value functions. 


1. Introduction 

Optimality conditions on control problems are usually constructed with 
the Pontryagin Maximum Principle (PMP) [28]. For infinite-horizon control 
problems, even in the free end-time case, the PMP can be degenerate mm- 

For infinite-horizon problems, one can often prove the nondegeneracy (the 
normality) of the PMP under additional assumptions. One may use uniform 
coercivity of running cost |29l Corollary 4.1], strong convexity of the Hamil¬ 
tonian |32] . or concavity of the dynamics function |29l Hypothesis 5.1]; or, 
one may impose asymptotic estimates on motions laiMi and costate arcs 
[21 Theorem 4],[T8l Remark 8],[T9l Proposition 4], e.g. their total variation 
[Ill2[5l|6l[29l[30]. The nondegeneracy of the PMP follows from the Lipschitz 
continuity of the value function (see [U Theorem 5.1], [20l [36|) under the 
Michel condition (see [22] ). Besides, the Lipschitz continuity of the value 
function is also guaranteed by estimates on motions and costate arcs e.g. 
their total variation, see [Il|2l|3l[5l[6l|29l[36|. 

Through Dynamic Programming Principle (DPP), we prove that, for a 
Lipschitz continuous value function, the normal form of PMP with sensi¬ 
tivity relations is a necessary and sufficient condition of optimality in view 
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of this value function. To make it a straightforward consequence of the re¬ 
sults [To] for finite horizon, we restate the optimality in terms of the value 
function. Choosing a corresponding value function, we obtain necessary con¬ 
ditions for various optimality criteria under different asymptotic constraints, 
including the optimality in weighted Lebesgue spaces [iESllSs], agreeable 
[laiiii, almost strong, and classical P[27] optimalities. In these results, we 
only assume the corresponding value function to exist and to be Lipschitz 
continuous. Generally, the same optimal control can correspond to several 
different value functions (see Example 17.31) . For weakly agreeable optimality 
criteria, we found a family of value functions parameterized by unboundedly 
increasing sequences. We also show the conditions on a control system (see 
Theorems I6.ll6.2p that guarantee the Lipschitz continuity for a finite value 
function without an assumption on the asymptotic behavior of trajectories 
or adjoint variables. 

We start with definitions, including the value function as a function satis¬ 
fying DPP and optimal control corresponding to a given value function. In 
the next section, we obtain a necessary (and sufficient) condition for such 
optimality. In Section (H we transfer these results to different asymptotic 
constraints. Next, we study value functions for agreeable optimality [13 El- 
Section [6] is devoted to the conditions on a control system that guarantee the 
Lipschitz continuity for a finite value function. The last section is devoted 
to examples. 


2. Problem Statement and Definitions 

The infinite horizon control problem. Consider the following optimal 
control problem for infinite horizon: 


(2.1) 

Minimize 

r+oo 

/ fo{t,x,u)dt 

Jo 

ue P, X E M”"; 

(2.2) 

subject to X = f{t,x,u), 

(2.3) 


x(0) = 6*. 


Let P be a complete separable metric space. As for the class of admissible 
controls, we consider the set of all Lebesgue measurable functions u : M>o ^ 
P that are bounded on every time compact and denote it by U. 

Assume that / : E>o x x P —>■ M™, /o : M>o x M™' x P —>■ M are contin¬ 
uous and locally Lipschitz continuous in x; also, let / satisfy the sublinear 
growth condition. Then, to each (6, t) E x M>o and each u E U, we can 
assign a solution of (j2.2l) with the initial condition x{t) = b. This solution 
is unique and it can be extended to the whole M>o; denote it by Xb,t,u- 
On the dynamic programming principle. 

For each 6* > 0, y E M™, T > 6, and u E U, set 
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Let V'^{9,y) denote the infimum of the following problem in the interval 
[e,T]: 


{ Minimize fo{t,x,u) dt 
subject to i; = f{t, x, u), u & P, x G M™; 

x{e) = y. 

Definition 2.1. For an interval I C M>o and a function V : I x R™ —>■ RU 
{—oo,+oo}, we say that V enjoys DPP (Dynamic Programming Principle) 
iff 


V(t,b) = inf 
neu 



•S, u(s))ds+F (r, 


V6 G R"*,[Lt] C /. 


Remark 2.2. For every T > 0, : [0,T] x R™ —>■ R satisfies DPP. 


On conditions of optimality. 

Note that the improper integral in (I2.1j) may not exist; as a consequence, 
for control problems on infinite horizon, there are several optimality criteria 
P[IIl[l3lll6l[l2lll7l[27l|37]. Hereinafter denote by u* the optimal control; 
however, we will always specify which criterion do we use for u*. Set also 

X = ■ 

Definition 2.3. Let V : R>o x R”* —>■ RU{—oo, +oo} be a function enjoying 
DPP. We will say that rt* G U is optimal in view ofY iff, for all T > 0, 


(2.4) V(r,x*(r)) + J(0,x*(0);u*,T) = V(0,x*(0)). 


3. Necessary and sufficient conditions in terms of value 

FUNCTIONS. 

Consider a Lipschitz continuous function /i : R^ ^ R and a point x G R^. 
The Frechet superdifferential d^h{x) is the set of vectors G R^ that satisfy 


lim sup 

y^x 


h(y) - h{x) - C{y - x) 

\\y - ^11 


< 0 . 


The limiting superdifferential d~^h{x) of /i at x consists of all () in R^ such 
that there exist sequences of x„ G R'’,Cn € d~^h{xn) satisfying Xn x, 
h{xn) —>■ h{x), Cn C- Foi' different (equivalent) definitions of the limiting 
superdifferential, see [9]. 

The Pontryagin Maximum Principle. 

Let the Hamilton-Pontryagin function H be given as follows: 


H{x,u,il),\f) := iff{t,x,u) - Xfo{t,x,u). 
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Let US introduce the relations 

(3.1) - ^(s) = —(x*(s),u*(s),V'(s), A,s), 

(3.2) sup Lf(x*(s), u, V'(s), A, s) =iL (x*(s), u*(s),-0(8), A, s) a.e, 

vGP 

(3.3) -V>(0)g5+V(0,x*(0)), 

(3.4) (Lr (^x*{s),u*{s),^p{s), 1, s) V'(s)) G c1’''V(s, x*{s)) a.e. 

Here, d^Y is the limiting superdifferential of a map 9 x V(s, x) G M. 

Halkin m proved that the Pontryagin Maximum Principle is a necessary 
condition of optimality for infinite-horizon problems: if an admissible u* G U 
is finitely optimal m for problem (I2.1jl - (l2.3jl . then there exists a nontrivial 
solution (iIj*, A*) of system (j3.ip - (l3.2jl for a.a. s > 0; here, A* G {0, 1}. 

Remember that, for the corresponding control problem with finite horizon 
T, under rather general conditions on the system, see, for example dm EH 
I35j . the Pontryagin Maximum Principle with A = 1 plus sensitivity relations 
(j3.3p - ()3.4jl forms a necessary and sufficient condition of optimality. Similar 
conditions arise in Hamilton-Jacobi-Isaacs PDEs for infinite horizon, see 

Thereinafter in this section we also assume that / and /o are differentiable 
with respect to x. 

Necessary conditions in terms of value function. 

Theorem 3.1. Assume a locally Lipschitz continuous function Y : M>o x 
M™’ —)> M satisfies DPP. Let u* be optimal in view ofY, i.e., satisfy UM- 
Then, there exists a co-state arc G C'(M>o,K™') satisfying PMP relations 
with A = 1 and sensitivity relations 

Proof Fix T > 0. Consider the following Bolza problem: 

{ Minimize fo{t, x, u) dt -\-Y{T; x{T)) 
subject to X = f{t, x,u), u ^ P, 

x(0) = b. 

Thanks to DPP, this problem has an optimal value, Y{0,b). Now, by (j2.4h . 
the control u* achieves the minimum of this problem, i.e., u* is optimal for 
it. 

Thanks to m Theorem 6.1], for some co-state arc G (^([O,T],M™), 
PMP relations (I3.ip - ()3.2p with A = 1 hold; moreover, satisfies (|3.4p (for 
a.a. s G [0,T]) and (13.3p . So, for every unbounded sequence of positive T„, 
co-state arcs with A = 1 satisfy (I3.1I) ~ (I3.2I) and relations (I3.3ll . (|3.4p for 
a.a. s G [0, T^j. 

Remember that Y is Lipschitz continuous, therefore c?j!'V(0,x*(0)) is a 
compact (see m- Consider the sequence of By (13.3h . this se¬ 

quence has a limiting point Q* G d,^V(0, x*(0)). Consider a solution 
G C(M>o,M™') of (13.ip with = C*! it satisfies (13.3p . By the theorem 
on continuous dependence of a differential equation on its initial conditions. 
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this solution is a partial limit of in the compact-open topology (on each 
time compact). Passing to the partial limit for a.a. positive s, one can 
provide that the co-state arc ijj* satisfies (I3.1j) ~ (j8.2j) and sensitivity relation 
(j3.4p for a.a. positive s. □ 

Sufficient conditions in terms of valne function. 

Theorem 3.2. Assume that a locally Lipschitz continuous function V : 
M>o X —)• M satisfies DPP. Let some co-state arc if G C'(M>o,M™') satisfy 
PMP relations (I3l])-(I321) with A = 1 and sensitivity relation (1331). 

Then, u* is optimal in view ofY, i.e., satisfies (Eap. 

Proof Fix T > 0. Consider Bolza problem (13.5|) . By DPP, this problem 
has an optimal value: V(0,6). Now, (12.41) holds for T iff u* achieves the 
minimum of this problem, i.e., if u* is optimal for this Bolza problem with 
b = 6*. 

By m Theorem 6.2], u* is optimal in this Bolza problem with b = 
iff there exists a co-state arc if'^ G (7(10,T],M™) with A = 1 that satisfies 
(I3.ip - ()3.2p and sensitivity relation (|3.4p for a.e. s G [0,T]. Set ijF = V'*|[o,t]- 
Since the co-state if* with A = 1 satisfies ()3.ip ~ ()3.21) . (13.41) . we obtain (I2.4p 
for all r > 0. □ 

Note that, in these theorems, the local Lipschitz continuity of V may hold 
not in M>o x MY but in a neighborhood of the graph of x*. In particular, 
one can require V to be locally Lipschitz continuous in a strongly invariant 
(for ()2.2p l neighborhood of the graph of x*. 

Of course, one would like to relax the condition of Lipschitz continuity of 
the value function. On the other hand. Theorem 16.11 provides this based on 
the finiteness of the value function if the optimal-time function [34] for this 
control system is Lipschitz continuous. 

4. Value Functions under Asymptotic Constraints. 

Asymptotic Constraints. 

For controls u', u" G U, for each T G M>o, the concatenation u' ot u' 
is as follows: (tt'o,-n")(t) := u'{t) if t < r, and [u'OT-u"){t) := u"{t), if f > r. 

Definition 4.1. We say that a multi-valued map x M>o ^ U 

induces asymptotic constraints iff, for all (6, t) G x M>o, 

(4.1) Do(^, t) = {uotUi : u £ U, ui G Qo{xb,t,uiT),T)}, MT > t. 

It is easy to see that each of the constant multi-valued maps U, B(R>o, P), 
Lp{M>Q, P) n U (if 1 < p < -|-oo) induces asymptotic constraints. 

Let us offer another example of asymptotic constraints. Fix a nonempty 
set M C M™. For all (6, t) G xM>o, denote by FtM{b, t) the set of all tt G U 
such that Q{xby,u{s), M) ^ 0 as s —)• -|-oo. Necessary conditions of optimality 
for problems under varying choice of M were considered, for example, in 
[Ml ESI ; for exit-time control problems, see [24|. Let us check that Dm does 
also induce asymptotic constraints. Indeed, for all (t, b) G M>o x M™, T > t, 
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u,ui G U, both the statement uot ui G and the statement 

ui G Q,Mixb,t,u{T),T) are equivalent to the fact that g{xbi,T,uiis), M) —0 
as s ^ +oo; here, bi := Xb,t,u{T)- Thus, Qm induces asymptotic constraints. 
On conditions of optimality nnder asymptotic constraints 
Dehne : M>o x M”* —)• M U {—oo,+oo} as follows: for all {t,b) G 
1R>0 X set V'^{t,b) = +oo if = 0; otherwise, 

V^(t,b) := inf liminf J(t, 6; u, T). 

^—^+00 

Theorem 4.2. Assume that induces asymptotic constraints. Suppose 
that the function is finite. Then, 

(1) enjoys the Dynamic Programming Principle (DPP); 

(2) u* is optimal in view ofV^ if u* lies in 0^(6*, 0) and satisfies 

(4.2) liminf J(0,6,; u*,T) = V%0,b^). 

T-^+cxd 

Proo/Note that, since is everywhere finite, flo is everywhere nonempty. 
By the dehnition of V^, for all r > 0, t G [0, r[, b G M™, we have 


V^{t,b) = inf liminf J{t,b\u,T) + J{T,Xbtu{x)'-,u,T) 
uen«(fe,t)T^-+oo L 


gr} 


inf 

u'eU,u''Gno(x^_j „,(r),T) L 


J(t, 6; u' o u", t) 


+ lim inf J(r, Xb,t,u'<>u" (t); ou'',T) 

T->+oo 


= inf 
u'eu 


J(t, 6; u', t) 


+ inf liminf J(r, u'hT) 

u"€n.{x,,.{r),r)T^+c^ ^ ^ ^ 


= inf 
u'GU 


J (t, b- u',t) + y (t, Xb^t,u’ (p)) 


Assume (|4.2p holds for u* G Do{b*,0). Then, for some unbounded se¬ 
quence of positive Tn, we have lim J(0, 6*; u*, r^) = 14^(0, 6*). Now, 



lim 

n^oc 

[J(0, 

b*] 

u* 

= 

J(0, 

6* ; u*, 

,T) 

+ 

gr} 





> 

J(0, 

6* ; u*, 

,T) 

+ 

> 

o' 

6* ; u*, 

,T) 

+ 


neOo(x*(r),T) n^oo 

jrO/n-: V7 


Thanks to the Dynamic Programming Principle, it implies (12.4h for V = 

□ 

On conditions without asymptotic constraints. 
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Define the function from M>o x to M U {—oo, + 00 } as follows: 

b) = inf liminf J{t, b; u, T) V(t, b) € M>o x R”*. 

neU T—>-00 “ 


One could easily prove that if Do := U. Now, Theorems 13.11 

and 14.21 imply the corresponding necessary conditions for u*. 

Corollary 4.3. Assume that f and fo are differentiable with respect to x. 

Let be finite and locally Lipschitz continuous. Let n* G U satisfy 

liminf J(0, 6,; m*,T) = 

T->+oo 

Then, there exists a co-state arc if G C'(R>o,R"*) satisfying PMP relations 
p.lll - p.2p with A = 1 and sensitivity relations (|3.3p0|3.4p for V = . 

Note that u* is optimal in view of if u* is an overtaking optimal 
control mm Therefore, we obtain necessary conditions of overtaking 
optimality. These necessary conditions, including the effective transversal- 
ity condition at infinity, can be found in mm- However, the necessary 
conditions in mm exploit the assumptions that are not required for Corol¬ 
lary (j4.3p : however. Corollary 14.31 assumes the value function to be known. 

Analogously, set Do = Lp(R>o,T’) n U. If u* achieves 14^(0, 6*), (j3.3p 
holds. For the linear case, this sensitivity relation was proved in [H (7.13)]. 

On almost strong and classical optimalities. 

Recall the optimality criteria from [8] and [27]. For each {b,t) G R™' x R, 
denote by (Qcibff)) the set of all u G U such that a map [t, -|-oo[9 

s fois, xi,^t,u{s), u{s)) has an improper Riemann (Lebesgue) integral. We 
claim that Ll-ji induces asymptotic constraints. Fix {b,t) G R™' x R>o, 
u,ui G U, and T > t. Set bi := Xh^t,uiT)- By definition, ui G Pt.'ji{bi,T) 
iff, for all T > 0, there exists a limit of /o('S, a;6i,r,«i(s), mi(s)) ds, i.e., if, 
specifically, fo{s,Xb^^T,uoTUi{s), {u oj- ui){s)) ds has a limit as r —>■ -|-oo. 
By Xbi,T,uoTUi = Xb,t,uoTUi, we obtain “D” in (jiT]) for Do = Dt^. On the 
other hand, u G Pt.'ji{bi,T) if u G QTz{b,t); thanks to u = u ot u, we also 
obtain the converse inclusion. The proof for Lie is similar. 

Let the mappings from R>o x R™' to RU {— 00 , -|-oo} be with 

Do = D^, Do = LIti, respectively. Under the assumption U^(0,x*(0)) G R, 
it is easy to verify that u* is classical optimal [8l Definition 7.5], [271 (LI)] iff 
u* G D£(a:*(0),0) holds (|4.2I) for U* = V^. Similarly, under the assumption 
U^(0, 6*) G R, u* is almost strongly optimal [H Definition 7.8], [TT] (Rl)] iff 
u* G D7^(x*( 0),0) holds (|4.2p for = V^. Theorems 13.11 and 14.21 imply 
necessary conditions of optimality for these criteria. In [U Sect. 7], see 
rather general conditions of existence for such optimal controls. 
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5. On Agreeable Control 


Definition 5.1. Call ti* E U a weakly agreeable control m Definition 
3.2(iii)] iff this control (with its motion x*, x*(0) = b^,) satisfies 

hminf ^ J(O, 6*; t) + (t, — inf J(0, 6*; u, T)^ <0, Vt > 0. 

Definition 5.2. Call u* E U an agreeable control m Definition 3.2(ii)] iff 
this control (with its motion x*, x*{0) = 6*) satisfies 

lim fj(0, 6*; n*, t) + x*(t)) — inf J(0, 6*; n, T)) < 0, Vt > 0. 

T—>+00 V iiSU / 

Lemma 5.3. For a control n* E U with its motion x*, u* is weakly agreeable 
iff there exists an unbounded sequence of positive Tn such that, for all T > 0, 

(5.1) J(0,x*(0);n*,r) = lim ri/"'"(0,x*(0)) - l/^"(r,x*(r))l 

n—foo 

Proof Let n* E U be weakly agreeable. Then, for a natural n, we can 
choose Tn > n such that J(0,x*(0);n*,n) + D'^"(n, x*(n)) — D'^"(0,x*(0)) < 
1/n. For all T E [0, n], the relation n* U C u* ot U holds. Hence, 

J(0,x*(0);n*,r) + !/"■"(T,x*(T)) < J(0,x*(0);n*,n) + H"'"(n,x*(n)), 

also, by DPP, 0 < J(0, x*(0); n*, T) + V'^"{T,x*{T)) — H’"’* (0, x*(0)) < 1/n. 
Passing to the limit as n —>■ 00 , we obtain (15.ip for this sequence of r^. 

The converse implication is clear. □ 

Necessary conditions of weakly agreeable optimality 


Corollary 5.4. Letu* E U 5e o weakly agreeable control in (12.ip ~ (j2.3p . i.e., 
for some unbounded sequence Tn t 00 , let u* satisfy (j5.ip for all T > 0. 

Let a function V°° : M>o x M U {— 00 , + 00 }, defined as 

(5.2) V°^{t,b) := {t,b), y{t,b) E M>o x R™, 

n^oo 

be finite and locally Lipschitz continuous. 

Then, there exists a co-state arc if E C(R>o, R™) such that PMP relations 
(j3.ip - (j3.2|] for A = 1 and sensitivity relations (13.31) - (13.4|] for V = hold. 


Proof Fix T > 0, {t,b) E [0,T] x R™. By Remark 12.21 satisfies DPP. 
Passing to the lower limit as n —)• 00 , we have 

V°°{t,b) = inf [ / fQ{s,Xbg,uis),u{s))ds + V°°{T,Xb,t,u{T)) 
uei] I Jt ” J 

for all r > 0, {t,b) E [0,r] x R”*. Thus, 1/°° satisfies DPP. 

Fix T > 0. By (j5.1l) . there exists a converging to 0 sequence of such 
that 

J(0, X* (0) ■,u*,T) + (T, X* (T)) = (0, x* (0)) + Sn 

holds. Passing to the lower limit as —>■ 00 , we obtain 


J(0, x*(0); n*, T) + V°°(T, x*(T)) = P°°(0, x*(0)), VT > 0. 

Now, Theorem [Q with V = V°° completes the proof. 

Sufficient conditions of weakly agreeable optimality 


□ 
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Lemma 5.5. Let r be an unbounded sequence of positive numbers. Let the 
value y°°(0, 5*), defined in (15.2 j) . be finite. Let u* satisfy ()2.4[1 /or V = . 

Then, u* is weakly agreeable, and (j5.1jl holds for some subsequence of t. 

Proo/Note that, by (j2.4p . the finiteness of F°°(0,x*(0)) implies that 
V^{t,x*{t)) is the same for all t > 0. Now, there exists a subsequence 
of := Tn(k) such that 14P(A:, x*(fe)) < V°°{k,x*{k)) + l/k for a natural k. 
Fix these r^. 

For each /c G N, T G [0, /c], we have u* Oj^lJ C u* U. Hence, 

J{0,b,,-u*,T) + V^k{T,x*{T)) < J{0,b^-,u*,k) + V^k{k,x*{k)) 

< J{0,b^;u*,k) + V°^{k,x*{k)) + l/k 

^ F°°(0,x*(0)) + 1/A: 

= V°°(T,x*{T)) + J{0,b,-,u*,T) + l/k. 

Thus, {T, X* {T)) converges to V°°{T,x*{T)) for all T > 0 as A; —)• oo. 

Now, for each T > 0, passing to the limit, we obtain 

0 ^ J{0,x*{0);u*,T) + V°°{T,x*{T))-V°^{0,x*{0)) 

= lim rj(0,x*(0);u*,T) + H"'fc(r,x*(r))-H"'fe(0,x*(0))l. 

fc->-oo ^ 

Thus, (|5.ip holds for r', and, by Lemma [5l3l u* is weakly agreeable. □ 
Applying Lemma 15.51 and Theorem 13.21 we obtain a sufficient condition 
for a weakly agreeable control. 

Corollary 5.6. For some unbounded sequence r / oo, let V°° be finite and 
locally Lipschitz continuous. Let a co-state arc G C'(M>o,M™') satisfy PMP 
relations (j3.1[) - ()3.2p with A = 1 and sensitivity relation (j3.4p forY = 1A°°. 
Then, (j5.ip holds for some subsequence of t, and u* is weakly agreeable. 

See the sufficient conditions of agreeable optimality in terms of the as¬ 
ymptotic behavior of V'(s)(x(s) —x*{s)) for large s in [121 Theorems 2.5 and 
2.6]. Such conditions are often used in proofs of the turnpike property [37] . 
Conditions for agreeable optimality. 

Theorem 5.7. Let a function defined as follows, 

(5.3) hm V^it,b), V(t, 5) G ]R>o x M™, 

T—>-+oo ~ 

be well-defined and locally Lipschitz continuous. 

Then, the following conditions are equivalent: 

(1) u* is weakly agreeable; 

(2) u* is agreeable; 

(3) there exists a co-state arc if G C'(M>o,M™') satisfying PMP relations 
(|3.ip - (|3.2p with A = 1 and sensitivity relations (13.3p , (|3.4I) for Y = 

-^all 
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Proof 2) => 1) was proved in [HI Proposition 3.2], 

1) 3). By Lemma [5?3l u* satisfies (j5.1|) for some unbounded sequence of 

positive Tn- Note that V°° = and all conditions of Corollary 15.41 hold. 
Therefore, for some co-state arc if € M™), relations (I3.1|) - (l3.2p for 

A = 1 and relations (j3.3p ~ (l3.4l) for V = hold. By V°° = 1) ^ 3) is 

proved. 

3) ^ 2). Assume the contrary. Then, by the definition of agreeable con¬ 
trols, there exist t > 0 and an unbounded sequence of positive Tn such that 

liminf (j{0,K;u*,t) + > 0. 

n—)-cxD \ / 

On the other hand, V°° = and all conditions of Corollary 15.61 hold, 
and (15.11) holds for u* with every unbounded sequence of positive numbers. 
However, it contradicts the choice of r^. □ 

Some conditions of validity of were shown in |14l Theorem 3.3]. In 
this case, holds |14l (3.13)]. Then, a weakly agreeable control is 

also weakly overtaking optimal m- Conditions |29l Hypothesis 3.1(i)-(iv), 
Hypothesis A.l] guarantee the Lipschitz continuity for well-defined V°° and 
(it is sufficient to repeat the proof of [291 Theorem A.l] verbatim). 
On the other hand, in Example 17.31 the value functions and P“^^ are 
Lipschitz continuous and there exists a control that is optimal for both value 
functions, however, P™^ > P“^^ holds everywhere. 

6. On Conditions of Lipschitz Continuity of Value Functions 

Usually, the Lipschitz continuity of a value function is guaranteed by 
asymptotic conditions on f,fo,J or on solutions of (I22|),(I3T]) , see [nisi El 
[5l m [291 ES] . We will use the Lipschitz continuity [Ml ES] of the optimal-time 
function of the control system. Let us obtain the conditions to guarantee 
the Lipschitz continuity of the value function in absence of any asymptotic 
conditions besides the finiteness of this function. 

Consider nonnegative integers r,s (r -|- s = m), a set W C M'’, and a 
compact subset P' C P. Consider optimal-time function Qw) optimal-time 
function under additional condition u{t) G P'(Vf > 0), and maximal¬ 
time function Q'^ as follows: for all y' = {w',z') & G,z ^ W,t' > 0, 

Qwit',y',z) :=inf{r >0 : 3 u £ U D B{R>o, P'),Xy',u{t'+ t) € Wx{z}}, 
Qw{t',y',z) :=inf{r >0 : 3 u £ U,Xy>^t',u(t'+T) £ Wx{z}}, 

Q'^{t',y',z) :=sup{t >0 :3u£ iJ,Xy>^t',u{t' +t) £ Wxjz}}, 

Q'^iy', z,t') := -|-oo if no motion joins {t',y') to [t',oo[xW x {z} under the 
requirement u(t) £ P' for all t > t', Q'^{y', z,t') := -|-oo if no motion joins 
{t',y') to [t',oo[xW X {z}, and Q^{t',y',z) := -|-oo if there exists a motion 
that does not join {t\y') to [T,oo[xW x {z}. 

Let I, W, and Z be non-empty open subsets of ]R>o, intW, respectively. 

We will prove the following theorems: 
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Theorem 6.1. Let V : M>o x —>■ M enjoy DPP. Assume that a function 

S' : Z —)• R and a positive locally Lipschitz continuous function i? : / x W — 
R>o satisfy 

V(t, {w, z)) = R{t, w)S{z) V(t, w, z) £ I X W X Z. 

For a point of I x W x Z, let there exist a neighborhood I' x W' x Z' of 
this point and a positive L such that 

Q'^{t', {w', z'),z) < L\\z' — z\\ Vt' G l', w' € W', z, z' G Z'. 

Then, V is locally Lipschitz continuous on L xW x Z. 

Theorem 6.2. Let V : R>o x R™ —>■ R enjoy DPP. Assume that a function 
S : Z ^ M. and a positive locally Lipschitz continuous function R : L xW ^ 
R>o satisfy 

V(t, {w, z)) = R{t, w)S{z) V(t, w, z) £ L X W X Z. 

For a point of L x W x Z, let there exist a neighborhood L' x W' x Z' of 
this point and a positive L such that 

\\z' - z\\/L < Qw(^^ iw',z'),z) < Q^ft', {w',z'),z) < L\\z' - zH 

holds for all t' £ L',w' £ W',z,z' £ Z'. 

Then, V is locally Lipschitz continuous on L x W x Z. 

Corollary 6.3. Assume that V : R>o x R™ —R satisfies DPP. 

Assume that 

0 G intconv {f{t,x,u) : u £ P'} \/{t,x) £ G 

holds for some open set G C R>o x R”^. 

Assume that a function S : R™ —)• R and a locally Lipschitz continuous 
function R : G ^ R>o satisfy 

Y{t,x) = R{t)S{x) V(t, x)gG'. 

Then, V is locally Lipschitz continuous in G. 

Corollary 6.4. Assume that V : R>o x R™ ^ R satisfies DPP. 

Assume that 

0 0 cl conv {f{t, x,u) : {t,x) £ G,u £ P} 

!holds! for some open set G C R>o x R™. 

Assume that a function S : R™ —>■ R and a locally Lipschitz continuous 
function R : G ^ R>o satisfy 

Y{t,x) = R{t)S{x) V(t, x)gG. 

Then, Y is locally Lipschitz continuous in G. 

Proof of Theorem 16.11 

Fix {t^,by/,hz) £ I xW X Z. Take a fitting neighborhood I' x W' x Z' of 
this point with some L > 1. It is safe to assume cl (/' x W' x Z') C I xW x Z 
to be compact, \L' be an interval!. It will suffice to prove that Y is Lipschitz 
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continuous in another, possibly smaller neighborhood of {t^,,bw,bz)- Set 
b := {bw,bz), G' := W' x Z'. 

Thanks to the sublinear growth of /, there exist a compact K (cl G' C 
int K C W X Z) and 7 > 0 such that y ^ G' implies 

Xy,T,uit + t) G K,t + T G I VneUn P'),t G I',t G [0, 7 ]. 

Now, we can choose M > max{L + 1, |V(t*, 6 )|} such that 
fo{t + T,y,u) < M, \R{t + T,w)\ G [1/M,M], 

\\f{t + T,y,u)\\ < M, \R{t,w') — R{t + T,w)\ < M\\w' — t(;|| + Mr 

hold for all y G K, {w,z), {w',z') G K Ci (W x Z),u G P',t G I,t G [ 0 , 7 ]. 
Now, we obtain \\xy^t,u{t + r) — y\\ < Mr for all y G G',t G I. Also, we 
have |5(6z)| < |V(t*, 6 )|/i?(t*, 6 vp) < Decreasing the neighborhood 

G' := W X Z' of b, we can assume diam Z' < ^jM. 

Fix t' G I. Consider w' G W',z',z'' G Z'. Set y' := (in',Z) E G'. By the 
choice of -L, we have 

(5^(('(c', z'), z"^ r) < L\\z' — z"\ I < M diam Z' < 7 . 

By the definition of Qw, we can choose tt E U n P'), w G W, and 

T > 0 enjoying 

Xy',t',u{t' + r) = (ih, z”), T < {L + l)\\z' — z''\\ < M diam Z' < 7 . 

By r < 7 and y' G G', we obtain (w, z") G K and 

(6.1) ||n) — re'll < \\xyi^t',u{^' + t) — y'\\ < Mr < M‘^\\z' — z"\\. 

Set t" := t' + r. By DPP, we have 

V(t',y') < J{t',y';u,f)+Y{f,Xy>^o^u{t”))- 
In view of V(t, (re, z)) = R{t, w)S{z) and /o(t, y, u) < M, we obtain 
R{t',w')Siz')=Y{t',y') 

< Mr + R{t'', iv)S{z") = Mt + V(t", {w, z”)). 


( 6 . 2 ) 


In particular, in the case y' := y,z" := bz, we can provide {w,z") G K, 
r < 7 , and 


R{t',w')S{z') < R{f,iv)S(bz) + MT<M\S{bz)\+M<2M^ 

by the choice of M. In the case y' := b, z" := z, we have {w, z) G K, t < 'y 
and, at last, 


R{t", w")S{z) > R{t', bw)S{bz) - tM > -tM> - 2 MA 

According to |i?| > 1/M, we also have | 5 ( 2 :)| < 2M^ for all {w,z) G G. 

Return to general case. Consider every w' G IF', z\ z" G Z' and, addi¬ 
tionally, every w” G W',t G L Then, we again obtain {w,P') G K and 
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T < 7 satisfying (j 6 .ip and (I6.2p . Now, by (j6.2|) . for all t,t' E I,w',w" E W, 
and z\ z" E Z'^ ,w\z') —Y{t,w", z") does not exceed 

Mr + {R{t'+T, w) — R{t', w') + R{t', w') — R{t, w"))S{z") 

< Mt + 2M^(r + ||tD — tc'll + + \t' — t|) 

EU 

< (M + 2M® + 2M®)r + 2M^{\\w" - re'll + \t' - t|) 

ED 

< 7M^{\\z' — -z"|| + lire" — re'|| + \t' — t\). 

Thus, 

\Y{t',w',z') -Y{t,w”,z")\ < 7M^(||z'-z"|| + lire"-re'll + |t'-t|) 

holds for all {t',w', z'), {t,w", z") from some neighborhood I x W x Z' of 
each (t*, bw, bz) G I x W x Z. □ 

Proof of Theorem 16.21 

Fix (t*, bw, bz) £ I xW X Z. Take a fitting neighborhood I' x W x Z' of 
this point with some L > 1. It is safe to assume cl (/' x W x Z') C I xW x Z 
to be compact, !/' be an interval!. It will suffice to prove that V is Lipschitz 
continuous in another, possibly smaller neighborhood of (t*, ftw, 6 z). Set 
b:= {bw,bz),G' :=}¥' X Z'. 

Thanks to the sublinear growth of /, there exist a compact K (cl G' C 
int K CW X Z) and 7 > 0 such that 

Xy^Y,uit T t) E Ki t -\- T ^ I V?/EG*,TE[0,7])'iiEUn i3(M>o, P ) ^ I ■ 

Now, we can choose M > max{L + 1, |V(t*, b)\} such that 
\fo{t + T,y,u)\ < M, \R{t + T,w)\ E [1/M,M], 

||/(t + T,y,u)\\ < M, \R{t,w') — R{t + T,w)\ < M||re' — rr)|| + Mr 

hold for all y & K, {w,z), {w' ,z') ^ K r\ (W x Z),uGP',tG I, [OjT]- 
Now, we obtain \\xy^t,uit + r) — y|| < Mr for all y E G',r E /. Also, we 
have 15 ( 62)1 < |V(t*, 6)|/i?(t*, 6w) < M^. Decreasing the neighborhood 
G' := W' X Z' of 6 , we can assume diam Z' < 7 /M. 

Fix t' E I. Consider w' E W, z', z" E Z'. Set y' := (re', z') E G'. Consider 
the following exit-time problem: 

{ Minimize fo{t, x, u) dt +Y{T; x{T)) 

subject to a: = f{t, x,u), u ^ P, 
x{t') = y', T = inf{t > t' : x{t) ^ W x {z”} U {-|-oo} 

By the definitions of Qw and for all rt E U, there exists such that 
t' + (re', z'), z”) <Tu = inf{t > t' : Xyi^t',u{b) ^ W x {z"}} 

< t' + QW(t',(re',z'),z"), 

i.e., 

(6.3) 0 < - t” < L||z'-/'ll, Xy/ 7 /,„(r„) E W X {/'}. 
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Since all are uniformly bounded (by t' + L\\z' — z"\\), the optimal value 
of this exit-time problem is in particular, 

V{t',y') < J{t',y'-u,Tu) + V{Tu,Xy'^t',ui'^u))- 

In addition, we can choose a control u' G U with its motion x' := Xy/^t'y 
such that 

\V{t\ y') - J{t\ y'-u, Tu) - V, x ’< ||/ - /'||. 

By (16.,ip we have x'{Tu') G W x {z"}. Now, we can choose tD G W such that 
x'{Tu') = {w,z"). So, 

(6.4) \R{t',w')Siy') - J{t',y';u,T^) - ,w)S{z')\ < ||z' - z"||. 

In addition, Tu' —t'<L\\z' — z"\ \ < M diam Z' < 7 . Then, by the choice of 
the compact K, we have x'{t) G K for all t G [t 'By the choice of M, 
we obtain 


(6.5) 


\w-w'\\ < \\xy>^t',u'{Tu') -y'W 

< M{Tu'-t') < M^\\z'-z" 


and I J(t',y';li,T„)| < M{Tu' —t'). Then, 

Ea 

( 6 . 6 ) \R{t', w')S{z') - R{Tu>,w)S{z")\ < 

E3J 

< 

< 


\\z' - z"\ \ + M{Tu> 

2M^\\z' -z"\\ 
2M7 < 2M^ 


t') 


In particular, in the case y' := y,z" := bz, we can provide {w,z") G K, 
r < 7 , and 

ESI 

\R{t',w')S{z')\ < \R{t'',iB)S{bz)\+2M^ < M\S{bz)\ + 2M^ <3M^ 


by the choice of M. According to \R\ > 1/M, we also have 
\S{z')\<3M^ y{w',z')eG'. 

Return to the general case. Consider every w' G W',z',z'' G Z' and, 
additionally, every w" G W',t G I. Then, we again obtain u' G IK, T^' G 
[t',t'+'y] and Xy' ,u'(Tu') = (w, z") G K satisfying (I6.5p . (l6.6p . Now, by (16.6h . 
for all t,t' G /, w',w" G IT', and z'^z" G Z'^ |V(f',M,2:') — V(f, tc", T')] does 
not exceed 


< 

E3J 

< 

< 


2M^\\z' -z''\\ + {R(Tu',w) - R(t',w') + R{t',w') - R(t,w''))S{z') 
M^\\z' — z''\ \ + 3M^(Tu' — t' -|- I|u) — rc'lI -|- — rc'|| -|- \t' — t|) 

M^\\z' - z"\\ + 3M®(1 + M)(r„/ - t') + 3M^{\\w'' - ic'|| + \t' - t|) 

7M'^\\z' - /'II + 3M^(\\w'' - u;'|| + |t' - t|). 
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Thus, \Y{t',w',z')-Y{t,w",z")\ < 7M'^{\\z'-z''\\ + \\w"-w'\\ + \t'-t\) holds 
for some neighborhood I x W' x Z' of each {ttf^bw',bz)&I'><WxZ. □ 

7. Examples 

Example 7.1. Let x be the capital stock, u the investment, u > 0 the 
depreciation rate, and the discount rate. Consider the following problem: 

r+oQ 

Minimize / e^^g{x,u)dt, 

Jo 

subject to i; = —i^x + u, x(0) = x* it G [0, Umax], k > 0. 

Set R{9) := for all 0 G M. It is easy to prove that 

J{e, y, u, T) = R{e)J{0, y;u',T-e) > 0, ?/ G M, T > 0, 

where u'{t) := u{t + 9) for all t > 0. Since it G fl iff G for each of 
G {n^, U}, we have 

V(0, y) = i?(0)V(O, y) Vy G M™, 0 > 0, V G {E^, E^, E““} 

if V is well-defined. 

Set Z< :=]0, Umax/i^{- We obtain 0 G —vx -|- int [0, Umax] for all x G .Z’<. 
By Corollary 16.21 V is Lipschitz continuous (if finite) in M>o x . 

Set Zy •.=\Umax/v,oo[. We obtain —vx -|- Umax < 0 for all x G Z>. By 
Corollary 16.41 V is Lipschitz continuous (if finite) in ]R>o x Z>. 

So, the value functions are Lipschitz continuous (if finite) in {(t, x) G 
M>o X (M>o I X 7 ^ Umax/^}', HO assumptions on g besides continuity are 
needed. 

Consider a use case of Theoremae 16.116.21 (compare with [38l |2T] ): 
Example 7.2. 

1*00 

Minimize / g{yi,y 2 ,u)dt 

Jo 

(7.1) subject to y'l = u, y 2 = yi, 

(2/i,y2)(0) = 6*, yi,y2,u£PcR. 

For a certain /c G M, let us also require 

g{uyi,u‘^y2,u) = u^g{yi,y2,u) Vi/ > 0, (yi,y2,^^) G 

Clearly, the functions from {E^, E*”-^, E^, do not depend on t. More 
to come. First, note that u G U iff all maps u'^{t) := u{t/i 2 ) are within U 
for all u > {). Fix {yi,y 2 ) G M?,u G U, u > 0. Set x := X(j^^ j^ 2 ),o,u ^nd 
x' := For all t >0, we have 

x{t) = x'{tlv), g{x'{tlv),u{tlv)) = v^g{x{t),u{t)), 

J(0, {vyuv‘^y2)]u'^,T/v) = {yi,y2);u,T). 

Now, for all finite V G {E^, E*”-^, E^, we have 

Y{t,vyi,u^y2) = v^~H{Q,yi,y2) Vi/ > 0,t > 0, (yi,y2) G 
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Define W := R>o, ^ ;= M. Let some V E be well- 

defined and finite. 

The case P = R. 

Consider G+ = {(yi, 1 / 2 ) £ R^ : 2/2 > 0} and the functions R : R>o x W —)• 
R>o and S' : Z —)• R defined as follows: 

(7.2) R{t,w) := w^~^,S{z) :=Y{0,z,l) Vz E R,ic > 0,t > 0; 
then, 

(7.3) V(t,yi,y 2 ) = 7?(t, v^)*S'(yi/v^) Vyi E R, y 2 , t > 0. 

In the coordinates {w := := 2 / 1 /y^), system (|7.ip has the form 

'll — z^ /2 

(7.4) 'w = zl2, z = -, M E R, {z,'w)^ZxW. 

w 

Since for every point ( 221 , 2 / 2 ) £ G+, in its sufficiently small neighborhood, 
the controls u := zLz‘^{'yi,y 2 ) provide for 2 : to increase/decrease with the 
speed at least 1/4 y/^. Set L := 4y/^; in view of Theorem 16.11 we find out 
that, in R>o x G+ (for y 2 > 0), the function V is Lipschitz continuous. The 
proof of the case y 2 < 0 is similar. 

Thus, in the case P := R, the function V is Lipschitz continuous under 

{( 2 / 1 , 222 ) GR^| 2/2 /O}. 

The subcase P = [-a^,a^], G® := {( 221 , 2 / 2 ) G R^ : 2/2 > 0, yf < 2 y2a^}. 
Consider the functions R : R>o x W —R>o, 5" : Z —R, defined by (17.21) . 
Then, V satisfies (17.311 . in the coordinates (in := y/y 2 ,z := yi/y^), system 
dm) has the form (17.41) . Note that, for every point (yi,y 2 ) E G“ , we have 
I 2 / 1 I < o,y/y 2 , i-e. z{yi,y 2 ) < a. Hence, in its sufficiently small neighborhood, 
the controls u := ± 2 ^(yi, y 2 ) E P provide for 2 to increase/decrease with the 
speed at least 1/4 y/^. Set L := 4y/^; in view of Theorem 16.11 we find out 
that, in R>o x G“ (for yf < a?‘y 2 ), the function V is Lipschitz continuous. 
The proof of the case —y^ > a^y 2 is similar. 

The subcase P = [-a^,o^], G\ = {(yi,y 2 ) E R^ : y 2 > 0,yi > 2o^y2}. 
Consider the functions R : R>o x W —>■ R>o and 5 ; Z —)• R defined as 
follows: 

R{t, w) := , S{z) := V(0, 1 , 2 ) V 2 E R, tc > 0, t > 0, 

then, 

V(t,yi,y 2 ) = R{t,yi)S{y 2 /yl) Vyi E R,y 2 ,t > 0. 

In the coordinates {w := yi, z := y 2 /yl), system (I7.ip has the form 

1 — ‘2iZU 

'll) = u, 2 =-, ( 2 , ic) E Z X W. 

w 

Note that, for every point (yi,y 2 ) E G“, we have yf > 2a?y2, i.e., 
2a?‘z{yi,y2) > 1. Hence, in its sufficiently small neighborhood, for every 
admissible control, the coordinate 2 strictly increases. Then, in view of 
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Corollary 16.31 we find out that, in ]R>o x (for yf > 2a?y 2 ), the function 
V is Lipschitz continuous. The proof of the case yf < 2a?y2 is similar. 

Thus, in the case P := [—a^,a^] the function V is Lipschitz continuous 
under 

{{yi^yi) G 12/2 7^ 0, yf / 2a^}. 


Example 7.3. Let us refine Example l7.lt the new example does not satisfy 
assumptions [29l Hypothesis 3.1(iv)], O Ch.3(A4)], [30l (C3)], [6l (H6)]. 

/•H-oo 


Minimize 

subject to X = 2rt — X, x(0) = 6*, u £ P 


[2rt + |ri| 


= [- 1 / 2 , 1 / 2 ], 


— x] dt 
X G M. 


Clearly, J{0,b;u,T) = Xb^^uiT) — b + ||'u||li([o,t],p) for all 6 G M, u G U, 
T > 0. Since Xbfl^u{T) > —2 holds for large T, the finiteness of the limit of 
J{0,b]u,T) as T —oo implies u G Li(M>o,T’), moreover, we have 


Xb,o,uiT) —0, 6 + J(0, 6 ; u, T) —p) > 0 as T —)• oo. 

Then, we obtain 5) = — 6 , i.e., tt* = 0 is optimal in view of 

Moreover, V^ = V'^ = 

But, for T > 1, we define a control tip G U as follows: ttp(f) = —1/2 for 
all t G [T—ln2,T], and ttp(f) = 0 otherwise. Then, we have Xbfl,UT(?) = be~^ 
if f G [0,r — ln2], Xbfi^uTi't) = e “*(6 + e'^/2) — 1 if t G [T — ln2,r]. Now, we 
obtain 

In 2 

Xb,o,uAT) = be-^ - 1/2, H^(0, b) < J(0, 6 ; up, T) = be-^ - 1/2 - 6 + —. 

One easily proves that V'^{0,b) = J(0, 6 ;up,r). Passing to the limit, we 
obtain b) = —b + for all 6 G M. Thus, 

V 6 G M. 

Since = const, (I2.4jl guarantees that u* = 0 is optimal in view 

of and at once. Moreover, u* = 0 is DH-optimal (in particular, 
agreeable) and overtaking optimal but is not strongly optimal |17j . 

Note that, in this example, the functions are well-defined and 

smooth. In m Proposition 3.2], for exit-time control problems, it was 
proved that if P“^^ is well-defined and continuous. Moreover, in 

this case P“^^ = P“^ is the unique nonnegative solution of the associated 
HJB equation. Conditions guaranteeing the continuity of P“*^ were showed 
in m Section 4]. 
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